Skip to content

Pull the package builder images from ghcr instead of building them - #1743

Merged
Desvelao merged 6 commits into
5.0.0from
feature/1707-prebuilt-builder-images
Nov 10, 2026
Merged

Pull the package builder images from ghcr instead of building them#1743
Desvelao merged 6 commits into
5.0.0from
feature/1707-prebuilt-builder-images

Conversation

@Ripdiegozz

@Ripdiegozz Ripdiegozz commented Nov 6, 2026 •
edited
Loading

Copy link
Copy Markdown

Description

Every Generate packages job builds its 2 builder images from scratch (the base image, plus the RPM or the DEB one): 3m21s to 8m36s per job, and more when the yum or apt mirrors are slow. The images only hold build tools.

This PR publishes the 3 builder images for amd64 and arm64 to ghcr, and makes build-packages.sh pull them. If an image can't be pulled, it builds it locally as today, so a registry outage or a missing image never fails a build. The design follows the review with DevOps in #1707.

This PR covers the builder images part of the second task of #1707.

Related to #1707

Proposed Changes

  • Images: ghcr.io/wazuh/pkg_dashboard_{base,rpm,deb}_builder_{amd64,arm64}, tagged <version>-<first 12 characters of the Dockerfile sha256>, for example pkg_dashboard_rpm_builder_amd64:5.0.0-5bed341dd031. dev-tools/build-packages/common/builder-image.sh computes the reference for the build and the workflows.
  • Inputs mounted at run time: the Dockerfiles no longer ADD the builder scripts, the spec, debian/, VERSION.json, plugins or run-with-retry.sh. build-packages.sh mounts them read-only, with a shared SELinux label, where the builders already read them. The Dockerfile is the only input of an image, so a version bump or a spec change doesn't need a new image.
  • Pull or build: build-packages.sh pulls the image (2 attempts). If the pull fails, it builds the image locally under the same reference and, in CI, prints a ::warning::. It never pushes. It still works locally without registry access.
  • Generate packages: the workflow gets packages: read, and the job logs in to ghcr with GITHUB_TOKEN. A failed login doesn't stop the job. Job names, outputs and log lines don't change.
  • Publish: 5_builderprecompiled_dashboard_builder-images-upload.yml runs on pushes to main and the version branches that change a Dockerfile, builder-image.sh or VERSION.json, and on workflow_dispatch. It builds each image on a GitHub-hosted runner of its architecture (ubuntu-24.04, ubuntu-24.04-arm) and pushes it with GITHUB_TOKEN. Tags are immutable: if a tag exists, it skips it. It never runs on pull requests.
  • Retention: 5_builderprecompiled_dashboard_builder-images-cleanup.yml runs weekly and on workflow_dispatch (with dry_run). For each package it deletes the untagged versions and the versions superseded by a newer tag of the same version, so every version branch keeps its current image. The rule is in common/builder-image-retention.jq, tested by common/test-builder-image-retention.sh.

Results and Evidence

Publish: 42288344785 published the 6 images with GITHUB_TOKEN, 2 to 3 minutes per image. The new workflow can't be dispatched until it's on main, so it ran from its push trigger on a temporary test branch. The tags are the ones 5.0.0 will compute after the merge, so the push run on 5.0.0 will find them and skip them.

Pull, on a temporary branch with this PR and #1737 merged, system: both:

Cell Pull of the 2 images
rpm x86_64 (42292063871) 20s
deb amd64 (42292063871) 30s
rpm aarch64 (42292067999) 41s
deb arm64 (42292067999) 54s

Building the 2 images took 3m21s to 8m36s per job before.

  • Every cell logs in to ghcr and pulls both images. None falls back to a local build.
  • The RPM cells pass Generate packages, Test package and Upload package.
  • The DEB cells fail after the pull, in dpkg-buildpackage: since Merge 4.14.12 into 5.0.0 #1740, the first entry of debian/changelog on 5.0.0 is 4.14.12-RELEASE, so the package is built as 4.14.12-0 and the builder doesn't find wazuh-dashboard_5.0.0-0_amd64.deb. The changelog comes from 5.0.0; this PR doesn't touch it.

Cleanup: tested only with gh stubbed. Its schedule and workflow_dispatch triggers only work once it's on main. The first run should use dry_run.

Local checks:

  • The 3 Dockerfiles build with an empty context.
  • With docker stubbed, every input is mounted where its builder reads it. A failed pull falls back to a local build and the script exits 0.
  • test-builder-image-retention.sh passes its 8 cases.
  • With gh stubbed, the cleanup deletes only the selected versions, dry_run deletes nothing, and a package that doesn't exist yet only prints a notice.

Artifacts Affected

None. The packages are built from the same inputs with the same tools.

Configuration Changes

6 new container packages in ghcr.io/wazuh, linked to this repository.

Documentation Updates

None.

Tests Introduced

dev-tools/build-packages/common/test-builder-image-retention.sh, for the retention rule.

Review Checklist

  • Code changes reviewed
  • Relevant evidence provided
  • Tests cover the new functionality
  • Configuration changes documented
  • Developer documentation reflects the changes
  • Meets requirements and/or definition of done
  • No unresolved dependencies with other issues
  • PR is linked to the relevant issue(s)
  • Correct labels applied (e.g., no-changelog)

@Ripdiegozz
perf(ci): Pull prebuilt package builder images from ghcr
d34e27e 
Mount the builder scripts, spec, debian/, VERSION.json, plugins and
run-with-retry.sh at run time, so each Dockerfile is the only input of
its image. Tag every builder image as
ghcr.io/wazuh/pkg_dashboard_<image>_builder_<arch>:<version>-<hash>,
where <hash> is the short sha256 of the Dockerfile.

build-packages.sh pulls that tag and builds it locally when the pull
fails, so a missing tag or registry access keeps today's behavior.
Generate packages logs in to ghcr with GITHUB_TOKEN to pull it.

Signed-off-by: Diego Garcia <diego.garcia@wazuh.com>
@Ripdiegozz
feat(ci): Publish and clean up the package builder images
bca2b42 
Upload the base, rpm and deb builder images for amd64 and arm64 to ghcr
when their Dockerfile changes on a version branch or main, or on demand.
A tag that already exists is never pushed again.

A weekly cleanup keeps the 3 latest versions of each image and deletes
untagged ones. A branch whose tag was deleted builds the image locally.

Signed-off-by: Diego Garcia <diego.garcia@wazuh.com>
@Ripdiegozz
fix(ci): Keep the current builder image of every version branch
62728b7 
- Cleanup: replace actions/delete-package-versions (keep the 3 latest) with
  a jq selection that deletes untagged versions and a tagged one only when
  all its tags are <version>-<hash> and none is the newest of its version.
  With 4 active version branches, "keep 3" deleted the image of one of them
  for good. A missing package now gives a notice instead of failing, and
  workflow_dispatch has a dry run. The filter has its own test script.
- Upload: run on VERSION.json changes, since the tag holds the version.
- build-packages.sh: try the pull twice before building locally, and warn
  in GitHub Actions when it falls back to the local build.
- Delete the Docker/.gitignore files of base, rpm and deb: they only listed
  files that build-packages.sh no longer copies into the contexts.

Signed-off-by: Diego Garcia <diego.garcia@wazuh.com>
@Ripdiegozz Ripdiegozz self-assigned this Nov 6, 2026
@Ripdiegozz
fix(ci): Grant ghcr read access at the workflow level
054b904 
The Generate packages job header also changes in the build-both-systems work, so granting packages: read for the whole workflow keeps both changes from conflicting. Jobs with their own permissions, like Test package, are not affected.

Signed-off-by: Diego Garcia <diego.garcia@wazuh.com>
@Ripdiegozz Ripdiegozz mentioned this pull request Nov 6, 2026
Closed
4 tasks
@Ripdiegozz
Ripdiegozz marked this pull request as ready for review November 6, 2026 22:40
@Ripdiegozz Ripdiegozz linked an issue Nov 6, 2026 that may be closed by this pull request
Reduce the dashboard package build time in the 5.0.0 Nightly #1707
Closed
4 tasks
@Ripdiegozz
fix(ci): Keep local package builds as before
ff3dacd 
Mount the repository inputs with a shared SELinux label (z) so other containers that mount the repository keep access to them, wait 5s instead of 15s between pull attempts, since a local build without a ghcr login always falls back to building the images, and restore the Docker context .gitignore files.

Signed-off-by: Diego Garcia <diego.garcia@wazuh.com>
@github-actions

github-actions Bot commented Nov 6, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ All unit and integration tests passing

🔗 Workflow run · commit f56261778dceb500b3aa5f410a47fdd9981af43f

@cristinavicognz cristinavicognz mentioned this pull request Nov 9, 2026
Closed
This was referenced Nov 9, 2026
Merged
Merged
rodrigofez
rodrigofez approved these changes Nov 9, 2026
View reviewed changes

@rodrigofez rodrigofez left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve: Generate packages pulls the 3 published images on amd64 and arm64 and saves 2 to 8 minutes per job; the build-locally fallback keeps every other path working.

Tested ff3dacd7ed85. CI-only change, no UI, so no browser test. Read the CI logs of the runs in the PR body, ran the retention test, shellcheck, actionlint, and the pull fallback with docker stubbed. Couldn't list the ghcr versions: the packages are private and my token has no read:packages.

Requirements (#1707, point 2)

  • 🟢 Publish base, rpm, deb for amd64 and arm64 to ghcr with GITHUB_TOKEN, from GitHub-hosted runners: 42288344785, 6 jobs on ubuntu-24.04 / ubuntu-24.04-arm.
  • 🟢 Tags <version>-<Dockerfile hash>, never overwritten: the Dockerfiles no longer ADD anything, so the hash covers every input of the image. The "already exists" skip hasn't run in CI yet.
  • 🟢 Generate packages pulls, else builds: all 4 cells log Using the published builder image; with the pull failing, the script builds the image and goes on.
  • 🟢 Weekly cleanup keeps each version branch's current image: 8 cases pass and the rule reads right. Not run against ghcr yet (it only runs from main).
  • 🟢 Images hold build tools only.

Pull vs build (Generate packages, time to get the 2 builder images)

Cell 5.0.0: build the 2 images PR: pull the 2 images
rpm x86_64 2m32s (42320462672) 20s (42292063871)
deb amd64 3m37s (42654740775) 31s (42292063871)
rpm aarch64 5m09s (42654757171) 41s (42292067999)
deb arm64 9m07s (42654746290) 54s (42292067999)

Before merging

  1. Rebase and rerun one DEB cell. The DEB cells failed on the old debian/changelog (built 4.14.12-0), which #1745 fixed on 5.0.0 after this branch was cut. dpkg-buildpackage did finish with the mounted debian/, so the mounts work, but no DEB run is green yet. The CI runs also used c473a49872 (:Z mounts, default retry delay), not the head.
  2. First cleanup run. dry_run defaults to false, and the first Monday after this reaches main deletes without a dry run unless someone dispatches one before. Low risk (only 1 upload has run, so each package should hold 1 version), but default: true would match the plan in #1707.
Checks
  • bash dev-tools/build-packages/common/test-builder-image-retention.sh: 8 passed.
  • shellcheck on the new scripts: 2 style notes (SC2001) in the test. build-packages.sh: no new warnings.
  • actionlint 1.7.7: the 2 new workflows are clean; 5 warnings in 5_builderpackage_dashboard.yml are on lines this PR doesn't touch.
  • builder-image.sh on the head gives 5.0.0-06292375c2b2 (base), 5bed341dd031 (rpm), f782f2999ea8 (deb): the tags the CI runs pulled.
  • git merge-tree with #1737 and with 5.0.0: no conflicts.
  • CI: Cypress jobs fail on every branch. Build and Verify on Linux (ciGroup1): 3126 tests passed, then a Jest worker crashed (SIGSEGV). Not related; this PR changes no JS.

@Desvelao
Desvelao merged commit c6dcd33 into 5.0.0 Nov 10, 2026
63 of 76 checks passed
@Desvelao
Desvelao deleted the feature/1707-prebuilt-builder-images branch November 10, 2026 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rodrigofez rodrigofez rodrigofez approved these changes

Assignees

@Ripdiegozz Ripdiegozz

Labels

no-changelog seasoned-contributor

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Reduce the dashboard package build time in the 5.0.0 Nightly

3 participants

@Ripdiegozz @rodrigofez @Desvelao